Image analysis method and image analysis apparatus

ABSTRACT

An image analysis method includes acquiring images of spatially different analysis regions. Each of the images of the analysis regions is constituted by pixels including a plurality of data acquired simultaneously or time-serially. The method further includes obtaining a cross-correlation between two analysis regions by using data of pixels of images of the analysis regions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/168,544, filed on Jun. 24, 2011, which is based upon and claims thebenefit of priority from prior Japanese Patent Application No.2010-145607, filed Jun. 25, 2010, which are hereby incorporated byreference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image analysis.

2. Description of the Related Art

An image analysis technique called fluorescence cross-correlationspectroscopy (FCCS) has been known. FCCS is disclosed in, for example,Michelle A. Digman, Claire M. Brown, Parijat Sengupta, Paul W. Wiseman,Alan R. Horwitz, and Enrico Gratton, “Measuring Fast Dynamics inSolutions and Cells with a Laser Scanning Microscope”, BiophysicalJournal, Vol. 89, pp. 1317-1327, August 2005. FCCS is designed toperform correlation analysis by continuously irradiating one or moremeasurement points in a sample with excitation light for a certainperiod of time (e.g., 10 sec) and detecting fluctuations in theintensity of fluorescence emitted from the measurement points, therebyestimating the number of molecules and a diffusion constant.

In addition, an image analysis technique called raster image correlationspectroscopy (RICS) is also known. RICS is disclosed in, for example,non-patent literature 2. RICS is designed to acquire a raster scannedimage(s) of one or more frames. A raster scanned image can be, forexample, a fluorescence image. The data of each pixel of a fluorescenceimage represents the information of the intensity of fluorescenceemitted from a corresponding point in a sample. The data of pixelsdiffer in acquisition time and position.

Correlation characteristics based on molecular fluctuations are obtainedby performing spatial autocorrelation analysis using the data of thesepixels. A diffusion constant and the number of molecules can be obtainedfrom the correlation characteristics of molecules. A molecular diffusiontime can be obtained from the diffusion constant. A molecular weight canbe obtained from the molecular diffusion time.

Since a molecular weight, the number of molecules, and the like can beevaluated by performing spatial autocorrelation analysis in this manner,it is possible to observe interactions between molecules.

In FCCS, since measurement points in a sample are irradiated withexcitation light for a relatively long period of time, the sample tendsto be damaged. In addition, a target that can be analyzed is limited toa sample with a short diffusion time, and this analysis technique cannotbe applied to a sample with a relatively long diffusion time.

In contrast to this, in RICS, since each point in a sample is irradiatedwith excitation light for a relatively short period of time, the damageof the sample is small. In addition, RICS can be effectively applied toa sample with a relatively long diffusion time.

Conventionally, an analysis based on RICS has been done to calculate thediffusion time of molecules (or the number of molecules) by using theimage data of one region. The analysis result is the evaluation of themotions of molecules in the region. Such an analysis cannot evaluate themotions of molecules between different regions.

BRIEF SUMMARY OF THE INVENTION

According to an aspect of an embodiment, an image analysis methodincludes acquiring images of spatially different analysis regions. Eachof the images of the analysis regions is constituted by pixels includinga plurality of data acquired simultaneously or time-serially. The imageanalysis method further includes obtaining a cross-correlation betweentwo analysis regions by using data of pixels of images of the analysisregions.

Advantages of the invention will be set forth in the description whichfollows, and in part will be obvious from the description, or may belearned by practice of the invention. Advantages of the invention may berealized and obtained by means of the instrumentalities and combinationsparticularly pointed out hereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the general description given above and the detaileddescription of the embodiments given below, serve to explain theprinciples of the invention.

FIG. 1 is a view schematically showing an image analysis apparatusaccording to the first embodiment of the invention;

FIG. 2 is a block diagram showing the functional blocks of a controlunit shown in FIG. 1;

FIG. 3 is a view showing an example of a fluorescence image of atwo-dimensional image;

FIG. 4 is a view showing an example of fluorescence images of athree-dimensional image;

FIG. 5 is a flowchart showing image analysis according to the firstembodiment of the invention;

FIG. 6 is a view showing an observation region and analysis regions setin the observation region;

FIG. 7 is a view showing an observation region and two analysis regionsbetween which a cross-correlation is obtained;

FIG. 8 is a view showing an image indicating the calculation result ofspatial cross-correlation values associated with molecules in a sampleby luminances;

FIG. 9 is a view showing the fitting result of the calculation resultsof the spatial cross-correlation values in FIG. 8;

FIG. 10 is a view showing an image indicating the calculation results ofspatial cross-correlation values associated with molecules in a sampleby luminances;

FIG. 11 is a view showing the fitting result of the calculation resultsof the spatial cross-correlation values in FIG. 10;

FIG. 12 is a view schematically showing an image analysis apparatusaccording to the second embodiment of the invention; and

FIG. 13 is a block diagram showing the functional blocks of a controlunit shown in FIG. 12.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments of the invention will be described below with referenceto the views of the accompanying drawing.

First Embodiment Apparatus Arrangement

FIG. 1 is a view schematically showing an image analysis apparatusaccording to the first embodiment of the invention. This image analysisapparatus is configured based on a scanning confocal optical, microscopefor fluorescent observation of a sample.

As shown in FIG. 1, an image analysis apparatus 100 includes a lightirradiation unit 110 to apply light, e.g., excitation light, to a sampleS, a photodetection unit 130 to detect light, e.g., fluorescence, fromthe sample S, a control unit 160 to perform necessary control for imageanalysis, and a sample stage 190 to support the sample S.

The sample S is contained in a sample container such as a microplate ora slide glass, and is placed on the sample stage 190. The sample stage190 supports the sample S so as to allow it to be moved in the lateraldirection (x and y directions) and height direction (z direction)relative to the light irradiation unit 110 and the photodetection unit130. For example, the sample stage 190 includes three stepping motorswhose output shafts are perpendicular to each other, so that the sampleS can be moved in the x, y, and z directions by these stepping motors.

The image analysis apparatus 100 is, for example, of a multiple lightirradiation/multiple light detection type. The light irradiation unit110 includes an n-channel light source unit 111. The photodetection unit130 includes an m-channel photodetection unit 131. The light source unit111 has n channels and can emit excitation light beams having ndifferent wavelengths. The photodetection unit 131 has m channels andcan detect fluorescence beams having m different wavelengths. The lightsource unit 111 does not necessarily have a plurality of channels, andmay have a single channel. In addition, the photodetection unit 131 doesnot necessarily have a plurality of channels, and may have a singlechannel.

The n-channel light source unit 111 of the light irradiation unit 110includes light sources 112 a, . . . , 112 n, collimator lenses 114 a, .. . , 114 n, and dichroic mirrors 116 a, . . . , 116 n. The lightsources 112 a, . . . , 112 n emit excitation light beams for excitingfluorescent dyes contained in the sample S to cause the sample S to emitlight beams (fluorescence beams). The wavelengths of the excitationlight beams emitted from the light sources 112 a, . . . , 112 ncorrespond to the types of fluorescent dyes contained in the sample Sand differ from each other. The light sources 112 a, . . . , 112 n areconstituted by, for example, laser light sources with oscillationwavelengths corresponding to the fluorescent dyes in the sample S. Thecollimator lenses 114 a, . . . , 114 n respectively collimate theexcitation light beams emitted from the light sources 112 a, . . . , 112n. The dichroic mirrors 116 a, . . . , 116 n respectively reflect theexcitation light beams passing through the collimator lenses 114 a, . .. , 114 n in the same direction. The dichroic mirrors 116 a, . . . , 116n respectively transmit excitation light beams striking them from theupper side in FIG. 1, and reflect excitation light beams striking themfrom the right side in FIG. 1. As a result, the excitation light beamswith different wavelengths, which are respectively emitted from thelight sources 112 a, . . . , 112 n, are combined into one beam afterpassing through the dichroic mirror 116 a. The dichroic mirror 116 nneed not transmit any excitation light, and hence may be replaced with asimple mirror.

The light irradiation unit 110 further includes a dichroic mirror 122, agalvano mirror 124, an objective lens 126, and an objective lens drivingmechanism 128. The dichroic mirror 122 reflects excitation light fromthe light source unit 111 toward the galvano mirror 124, and transmitsfluorescence emitted from the sample S. The galvano mirror 124 reflectsexcitation light toward the objective lens 126 and changes itsreflecting direction. The objective lens 126 converges excitation lightand irradiates a measurement point in the sample S with the light, andcaptures light from the measurement point in the sample S. As theobjective lens 126, a lens having a large NA (Numerical Aperture) isused to form a minute confocal region (measurement point). The confocalregion obtained by this lens has a generally cylindrical shape with adiameter of about 0.6 μm (in an x-y plane) and a length of about 2 μm(in the z direction). The galvano mirror 124 constitutes an x-y scanningmechanism to scan a measurement point in the x and y directions. The x-yscanning mechanism may be constituted by using an acoustooptic modulator(AOM), a polygon mirror, a hologram scanner, and the like instead ofbeing constituted by using a galvano mirror. The objective lens drivingmechanism 128 moves the objective lens 126 along the optical axis. Thismoves a measurement point in the z direction. That is, the objectivelens driving mechanism 128 constitutes a z scanning mechanism to scan ameasurement point in the z direction.

The photodetection unit 130 commonly includes the objective lens 126,the galvano mirror 124, and the dichroic mirror 122 with the lightirradiation unit 110. The photodetection unit 130 further includes aconvergence lens 132, a pinhole 134, and a collimator lens 136. Theconvergence lens 132 converges light transmitted through the dichroicmirror 122. The pinhole 134 is placed at the focus of the convergencelens 132. That is, the pinhole 134 is placed at a position conjugate toa measurement point in the sample S, and selectively transmits lightfrom only the measurement point. The collimator lens 136 collimateslight passing through the pinhole 134. The light passing through thecollimator lens 136 strikes the m-channel photodetection unit 131.

The m-channel photodetection unit 131 includes dichroic mirrors 138 a, .. . , 138 m, fluorescence filters 140 a, . . . , 140 m, andphotodetectors 142 a, . . . , 142 m. The dichroic mirrors 138 a, . . . ,138 m selectively reflect light beams with wavelengths near thewavelength region of fluorescence to be detected, respectively. Thedichroic mirror 138 m need not transmit any light, and hence may bereplaced with a simple mirror. The fluorescence filters 140 a, . . . ,140 m respectively cut off unnecessary wavelength components of lightfrom the light reflected by the dichroic mirrors 138 a, . . . , 138 m,and selectively transmit only fluorescence beams generated by excitationlight beams from the light sources 112 a, . . . , 112 n. Thefluorescence beams transmitted through the fluorescence filters 140 a, .. . , 140 m respectively strike the photodetectors 142 a, . . . , 142 m.The photodetectors 142 a, . . . , 142 m output signals corresponding tothe intensities of the incident light beams. That is, the photodetectors142 a, . . . , 142 m output fluorescence intensity signals from ameasurement point in the sample S.

The control unit 160 is constituted by, for example, a personalcomputer. The control unit 160 performs acquisition/storage/display of afluorescence image of an observation region of the sample S, waits forthe input of settings of an analysis region, and performs image analysisprocessing (e.g., calculation of a correlation value and estimation ofthe number of molecules and a diffusion time). The control unit 160controls the galvano mirror 124 as an x-y scanning mechanism, theobjective lens driving mechanism 128 as a z scanning mechanism, thesample stage 190, and the like.

FIG. 2 shows the functional blocks of the control unit shown in FIG. 1.As shown in FIG. 2, the control unit 160 includes a scan control unit162, an image forming unit 164, a storage unit 166, a display unit 168,an input unit 170, an analysis region setting unit 172, a dataextraction unit 174, an analysis processing unit 176, and a stagecontrol unit 180. The scan control unit 162, the image forming unit 164,the storage unit 166, and the stage control unit 180 constitute anobservation region image acquisition unit in cooperation with the lightirradiation unit 110 and the photodetection unit 130, which have beendescribed above. This observation region image acquisition unit furtherconstitutes an analysis region image acquisition unit in cooperationwith the analysis region setting unit 172 and the data extraction unit174.

The scan control unit 162 controls the galvano mirror 124 so as toraster-scan the irradiation position of excitation light on the sample Swhen acquiring a fluorescence image of the sample S. In addition, ifnecessary, the scan control unit 162 controls the objective lens drivingmechanism 128 so as to z-scan the irradiation position of excitationlight on the sample S. The image forming unit 164 forms a fluorescenceimage of the sample S from the information of the irradiation positionof excitation light input from the scan control unit 162 and outputsignals from the photodetectors 142 a, . . . , 142 m. With thisoperation, a fluorescence image is acquired. The storage unit 166 storesthe fluorescence image formed by the image forming unit 164. The displayunit 168 displays a fluorescence image of the sample S and an analysisprocessing result. The input unit 170 includes, for example, a mouse anda keyboard, and constitutes a GUI in cooperation with the display unit168. This GUI is used to set, for example, an observation region andanalysis regions. The stage control unit 180 controls the sample stage190 to set, for example, an observation region in accordance with inputinformation from the input unit 170. The analysis region setting unit172 sets analysis regions in accordance with input information from theinput unit 170. The data extraction unit 174 extracts the data of twoanalysis regions between which a correlation is obtained. The analysisprocessing unit 176 obtains a cross-correlation by using the data ofpixels of images of two analysis regions. The processing performed bythis analysis processing unit 176 will be described in detail later.

Referring to FIG. 1, the excitation light beams emitted from the lightsources 112 a, . . . , 112 n are applied to a measurement point in thesample S through the collimator lenses 114 a, . . . , 114 n, thedichroic mirrors 116 a, . . . , 116 n, the dichroic mirror 122, thegalvano mirror 124, and the objective lens 126. The measurement point towhich excitation light beams are applied is raster-scanned by thegalvano mirror 124 in the x and y directions. In addition, if necessary,the measurement point is z-scanned by the objective lens drivingmechanism 128 every time one raster scan is complete. The measurementpoint is scanned over an observation region. The sample S emitsfluorescence from the measurement point upon receiving excitation light.The light (including undesired reflected light in addition tofluorescence) from the sample S reaches the pinhole 134 through theobjective lens 126, the galvano mirror 124, the dichroic mirror 122, andconvergence lens 132. Since the pinhole 134 is placed at a positionconjugate to the measurement point, only light from the measurementpoint in the sample S passes through the pinhole 134. The light passingthrough the pinhole 134, i.e., the light from the measurement point inthe sample S, strikes the m-channel photodetection unit 131 through thecollimator lens 136. The light that has struck the m-channelphotodetection unit 131 is separated in accordance with the wavelengths(that is, spectroscoped) by the dichroic mirrors 138 a, . . . , 138 m,and undesired components are removed by the fluorescence filters 140 a,. . . , 140 m. The fluorescence beams passing through the fluorescencefilters 140 a, . . . , 140 m respectively strike the photodetectors 142a, . . . , 142 m. The photodetectors 142 a, . . . , 142 m respectivelyoutput fluorescence intensity signals indicating the intensities of theincident light beams, i.e., the fluorescence beams emitted from themeasurement point in the sample S. The fluorescence intensity signalsare input to the image forming unit 164. The image forming unit 164processes the input fluorescence intensity signals in synchronism withposition information in the x and y directions (also in the z direction)to form a fluorescence image of the observation region in the sample S.The formed fluorescence image is stored in the storage unit 166. Thefluorescence image stored in the storage unit 166 is directly displayedon the display unit 168 or processed by the analysis processing unit 176to display the analysis processing result on the display unit 168.

[Observation Region and Spatial Correlation Calculating Formula]

A fluorescence image of an observation region is constituted by pixelshaving a plurality of data acquired time-serially. In practice, ameasurement point has a spatial spread in the x, y, and z directions. Apixel has a size corresponding to the spatial spread of this measurementpoint. If the observation region is a two-dimensional region, thefluorescence image is a two-dimensional image in which pixels havingsizes in the x and y directions are arrayed two-dimensionally. If theobservation region is a three-dimensional region, the fluorescence imageis a three-dimensional image in which pixels having sizes in the x, y,and z directions are arrayed three-dimensionally. From a differentviewpoint, a three-dimensional image is constituted by two-dimensionalimages at different z positions.

FIG. 3 shows an example of a two-dimensional image. Referring to FIG. 3,reference symbol τ_(p) denotes a pixel time, the acquisition timedifference between a given pixel and the next pixel adjacent to thegiven pixel. That is, the pixel time τ_(p) is the time required toacquire the data of one pixel. Reference symbol τ_(l) denotes a linetime, the acquisition time difference between the first pixel on a givenline and the first pixel on the next line. That is, the line time τ_(l)indicates the time required to scan one line.

Equation (1) given below represents a spatial cross-correctioncalculating formula used for RICS analysis used for a two-dimensionalimage. Equation (1) is an example of a cross-correlation calculatingformula for an analysis region 1 and an analysis region 2.

$\begin{matrix}{{G_{s}\left( {\xi,\psi} \right)} = \frac{\Sigma \; {I_{1}\left( {x,y} \right)}*{{I_{2}\left( {{x + \xi},{y + \psi}} \right)}/M_{12}}}{\left( {\Sigma \; {{I_{1}\left( {x,y} \right)}/M_{1}}} \right)\left( {\Sigma \; {{I_{2}\left( {x,y} \right)}/M_{2}}} \right)}} & (1)\end{matrix}$

where G_(s) is a spatial cross-correlation value of RICS, I₁ is the dataof a pixel of an image of the analysis region 1, e.g., fluorescenceintensity data, I₂ is the data of a pixel of an image of the analysisregion 2, e.g., fluorescence intensity data, x and y are the spatialcoordinates of the measurement point, ξ and ψ are the changes of thespatial coordinates from the measurement point, M₁₂ is the number oftimes of sum-of-product calculation of the data of the analysis region 1and the analysis region 2, M₁ is the total number of data of theanalysis region 1, and M₂ is the total number of data of the analysisregion 2.

Equation (2) represents a fitting formula used for RICS analysis on atwo-dimensional image.

$\begin{matrix}{\mspace{79mu} {{{G_{s}\left( {\xi,\psi} \right)} = {{S\left( {\xi,\psi} \right)}*{G\left( {\xi,\psi} \right)}}}\mspace{20mu} {{S\left( {\xi,\psi} \right)} = {\exp \left( {- \frac{\frac{1}{2}*\left\lbrack {\left( \frac{2{\xi\delta}_{r}}{W_{0}} \right)^{2} + \left( \frac{2{\psi\delta}_{r}}{W_{0}} \right)^{2}} \right\rbrack}{\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi}} \right)}}{W_{0}^{2}}} \right)}} \right)}}{{G\left( {\xi,\psi} \right)} = {\frac{1}{N}\left( {\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi}} \right)}}{W_{0}^{2}}} \right)^{- 1}*\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi}} \right)}}{W_{z}^{2}}} \right)^{{- 1}/2}} \right)}}}} & (2)\end{matrix}$

where G_(s) is a spatial cross-correlation value of RICS, S is theinfluence of a scan in RICS analysis, G is the influence of a time delayin RICS analysis, D is a diffusion constant, δ_(r) is a pixel size, N isthe number of molecules, ξ and ψ are the changes of the spatialcoordinates from the measurement point, W₀ is the radius of anexcitation laser beam in the lateral direction, W_(z) is the radius ofan excitation laser beam in the longitudinal direction, τ_(p) is thepixel time, and τ_(l) is the line time.

FIG. 4 shows an example of a three-dimensional image. Referring to FIG.4, reference symbol τ_(p) denotes the pixel time; τ_(l), the line time;and τ_(f), a frame time, the acquisition time difference between thefirst pixel of a given frame and the first pixel of the next frame. Thatis, the frame time τ_(f) indicates the time required to scan one frame.

Equation (3) given below represents a spatial cross-correlationcalculating formula used for RICS analysis on a three-dimensional image.Equation (3) is an example of a cross-correlation calculating formulafor the analysis region 1 and the analysis region 2.

$\begin{matrix}{{G_{s}\left( {\xi,\psi,\eta} \right)} = \frac{\Sigma \; {I_{1}\left( {x,y,z} \right)}*{{I_{2}\left( {{x + \xi},{y + \psi},{z + \eta}} \right)}/M_{12}}}{\left( {\Sigma \; {{I_{1}\left( {x,y,z} \right)}/M_{1}}} \right)\left( {\Sigma \; {{I_{2}\left( {x,y,z} \right)}/M_{2}}} \right)}} & (3)\end{matrix}$

where G_(s) is a spatial cross-correlation value of RICS, I₁ is the dataof a pixel of an image of the analysis region 1, e.g., fluorescenceintensity data, I₂ is the data of a pixel of an image of the analysisregion 2, e.g., fluorescence intensity data, x, y, and z are the spatialcoordinates of the measurement point, ξ, ψ, and η are the changes of thespatial coordinates from the measurement point, M₁₂ is the number oftimes of sum-of-product calculation of the data of the analysis region 1and the analysis region 2, M₁ is the total number of data of theanalysis region 1, and M₂ is the total number of data of the analysisregion 2.

Equation (4) represents a fitting formula used for RICS analysis on athree-dimensional image.

$\begin{matrix}{\mspace{79mu} {{{G_{s}\left( {\xi,\psi,\eta} \right)} = {{S\left( {\xi,\psi,\eta} \right)}*{G\left( {\xi,\psi,\eta} \right)}}}\mspace{20mu} {{S\left( {\xi,\psi,\eta} \right)} = {\exp \left( {- \frac{\frac{1}{2}*\left\lbrack {\left( \frac{2{\xi\delta}_{r}}{W_{0}} \right)^{2} + \left( \frac{2{\psi\delta}_{r}}{W_{0}} \right)^{2} + \left( \frac{2{\eta\delta}_{r}}{W_{0}} \right)^{2}} \right\rbrack}{\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi} + {\tau_{f}\eta}} \right)}}{W_{0}^{2}}} \right)}} \right)}}{{G\left( {\xi,\psi,\eta} \right)} = {\frac{1}{N}\left( {\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi} + {\tau_{f}\eta}} \right)}}{W_{0}^{2}}} \right)^{- 1}*\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi} + {\tau_{f}\eta}} \right)}}{W_{z}^{2}}} \right)^{{- 1}/2}} \right)}}}} & (4)\end{matrix}$

where G_(s) is a spatial cross-correlation value of RICS, S is theinfluence of a scan in RICS analysis, G is the influence of a time delayin RICS analysis, D is a diffusion constant, δ_(r) is a pixel size, N isthe number of molecules, ξ, ψ, and η are the changes of the spatialcoordinates, W₀ is the radius of an excitation laser beam in the lateraldirection, W_(z) is the radius of an excitation laser beam in thelongitudinal direction, τ_(p) is the pixel time, τ_(l) is the line time,and τ_(f) is the frame time.

[Measurement Procedure]

An image analysis procedure will be described below with reference toFIG. 5. Each step will be described with reference to FIGS. 6, 7, 8, 9,10, and 11, as needed.

(Step S1)

A fluorescence image(s) of one or more frames of an observation regionof the sample S is acquired. A fluorescence image is acquired throughone channel of the light source unit 111 and one corresponding channelof the photodetection unit 131. An observation region is atwo-dimensional or three-dimensional region. Accordingly, a fluorescenceimage is a two-dimensional or three-dimensional image. In the case of atwo-dimensional image, the fluorescence image shown in FIG. 3corresponds to one frame. In the case of a three-dimensional image, thefluorescence images shown in FIG. 4 correspond to one frame. The data ofeach pixel of a fluorescence image is, for example, the intensity offluorescence emitted from the corresponding measurement point.

(Step S2)

As shown in FIG. 6, analysis regions are set for a fluorescence image(s)of each frame of the observation region. The analysis regions arespatially different regions, which are generally spaced apart from eachother and do not overlap. Each analysis region is a two-dimensionalregion for a two-dimensional observation region, and is generally athree-dimensional region but may be a two-dimensional region for athree-dimensional observation region. Note that FIG. 6 shows theinterior of one cell, and an elliptic portion A of a central portionindicates the nucleus.

(Step S3)

As shown in FIG. 7, two analysis regions (an analysis region 1 and ananalysis region 2) between which a spatial cross-correlation is obtainedare selected for the fluorescence image(s) of one frame of theobservation region. In this case, a region X inside the nucleus isselected as the analysis region 1, and a region Y outside the nucleus isselected as the analysis region 2.

(Step S4)

The data of pixels corresponding to the two analysis regions (theanalysis region 1 and the analysis region 2) are extracted.

(Step S5)

A correlation is calculated by using the extracted data of the pixels.The spatial cross-correlation calculating formula given by equation (1)is used for a two-dimensional image. The spatial cross-correlationcalculating formula given by equation (3) is used for athree-dimensional image.

The data of each pixel used for correlation calculation may be the dataof the pixel of interest itself or the statistical value of the data ofpixels including the pixel of interest. The pixels may include the pixelof interest and its adjacent pixels. A statistical value may be, forexample, any one of an average value, a maximum value, a minimum value,a relative difference, an absolute difference, and a relative ratio ofthe data of the pixels. Depending on which kind of information to beobtained by RICS analysis will determine which kind of statistical valueis to be used.

In addition, data to be used for correlation calculation may be any oneof a pixel time, a line time, a frame time, a pixel positionalrelationship, a pixel size, and its statistical value.

Correlation calculation may be performed for the images obtained byreconstructing images based on the data of pixels. For example, the dataof adjacent pixels are added to decrease the number of data of pixels tohalf. Alternatively, the data of one pixel is divided into parts.Although the number of data of pixels does not normally increase once animage is acquired, pixel data that cannot be normally obtained iscompensated by assuming that the intensities of the acquired data of apixel are distributed around the data of the pixel in a Gaussiandistribution. Although the number of data of each pixel does notessentially increase, the appearance of the image is improved.

(Step S6)

Fitting is performed for the correlation calculation result obtained instep S5 to estimate at least one of the number of molecules and adiffusion time. The fitting equation given by equation (2) is used for atwo-dimensional image, whereas the fitting equation given by equation(4) is used for a three-dimensional image.

More specifically, the cross-correlation values G_(s) corresponding todifferent delay times are obtained by using equation (1) or (3). Adiffusion constant and the number of molecules are then obtained fromthe relationship between the cross-correlation values G_(s) and thedelay times by using equation (2) or (4).

In practice, the number of molecules and a diffusion constant areobtained by a residue comparison (fitting) between the results ofcorrelation calculation of the theoretical values of equations (2) and(4) and measured values. In fitting, first of all, (a) G_(s) obtained asa theoretical value (to be referred to as the theoretical correlationvalue G_(s) hereinafter) is calculated by using the predetermineddiffusion constant D and the number N of molecules. (b) The theoreticalcorrelation value G_(s) is then compared with the correlation valueG_(s) obtained as a measured value (to be referred to as the measuredcorrelation value G_(s) hereinafter) to calculate the residue betweenthem. (c) The new theoretical correlation value G_(s) is calculated bychanging the diffusion constant D and the number N of molecules in thetheoretical correlation value G_(s). (d) The new theoretical correlationvalue G_(s) is compared with the measured correlation value G_(s) tocalculate the residue between them. (e) The residue obtained in (b) iscompared with the residue obtained in (d) to determine whether theresidue has increased or decreased. In this manner, in fitting, (b) to(e) are repeated while the diffusion constant D and the number N ofmolecules in the theoretical correlation value G_(s) are changed,thereby finally obtaining the theoretical correlation value G_(s) thatminimizes the residue between the measured correlation value G_(s) andthe theoretical correlation value G_(s). The diffusion constant D andthe number N of molecules in the finally obtained theoreticalcorrelation value G_(s) are the diffusion constant D and the number N ofmolecules in the measured correlation value G_(s). As described above,fitting based on equation (2) or (4) is to estimate the optimal numberof molecules or diffusion constant in a two-dimensional orthree-dimensional observation region while changing the diffusionconstant D and the number N of molecules in the theoretical correlationvalue G_(s).

Diffusion constants and diffusion times have the relationshiprepresented by equation (5) given below. A diffusion time can thereforebe obtained from an obtained diffusion constant.

τ=W ₀ ²/4D  (5)

(Step S7)

The image based on the number of molecules or diffusion constant isdisplayed and stored. FIGS. 8, 9, 10, and 11 each show an example of ananalysis result. FIGS. 8 and 10 each show an example of displaying thecalculation results of spatial cross-correlation values associated withmolecules in the sample S as an image on a CRT. FIGS. 9 and 11 each showthe fitting result of the calculation results of the spatialcross-correlation values in FIGS. 8 and 10. FIGS. 8 and 10 each indicatethe magnitudes of the spatial cross-correlation values by luminances onthe CRT. Note that, referring to FIGS. 8 and 10, contour linesexplicitly indicate changes in luminance (spatial cross-correlationvalue). In addition, FIGS. 8 (9) and 10 (11) differ in the combinationof two analysis regions. The analysis results shown in FIGS. 8 and 9indicate that the motions of molecules between the analysis region 1(the region X in FIG. 6) and the analysis region 2 (the region Y in FIG.6) are relatively fast. The analysis results shown in FIGS. 10 and 11indicate that the motions of molecules between the analysis region 1(the region X in FIG. 6) and the analysis region 2 (a region Z in FIG.6) are relatively slow.

The above steps complete the analysis processing of the image for acombination of two analysis regions. Steps S3 to S7 are repeated foranother combination of two analysis regions. Alternatively, steps S3 toS7 are repeated for another fluorescence images of observation regionsof another frame.

According to this embodiment, it is possible to evaluate the motions ofmolecules between different regions in the sample S. In addition,obtaining spatial cross-correlations while sequentially changing acombination of analysis regions allows to two-dimensionally orthree-dimensionally comprehend the motions of molecules betweendifferent regions in the sample S. For example, obtaining spatialcross-correlations between combinations of one analysis region andsurrounding analysis regions allows to two-dimensionally orthree-dimensionally comprehend the motions of molecules between the oneanalysis region and the surrounding analysis regions.

In this embodiment, fluorescence images are obtained by the scanningconfocal optical microscope. In this case, images are formedtime-serially, the time when a fluorescence image of the analysis region1 is obtained differs from the time when a fluorescence image of theanalysis region 2 is obtained. For this reason, fluorescence imagesobtained time-serially allow to evaluate the motions of molecules whosemoving speeds are relatively low. Note that when images are formedsimultaneously (when captured by a two-dimensional image sensing devicesuch as a CCD or a CMOS), the time when a fluorescence image of theanalysis region 1 is obtained coincides with the time when afluorescence image of the analysis region 2 is obtained. In this case,it is possible to evaluate the motions of molecules whose moving speedsare relatively high.

Second Embodiment

FIG. 12 schematically shows an image analysis apparatus according to thesecond embodiment of the invention. This image analysis apparatus isconfigured based on a scanning confocal optical microscope forfluorescent observation of a sample.

As shown in FIG. 12, an image analysis apparatus 200 includes two lightirradiation units 210A and 210B to respectively apply light, e.g.,excitation light, to a sample S, two photodetection units 230A and 230Bto respectively detect light, e.g., fluorescence, from the sample S, acontrol unit 260 to perform necessary control for image analysis, and asample stage 290 to support the sample S. The light irradiation units210A and 210B respectively apply light beams to different regions in thesample S. The photodetection units 230A and 230B respectively detectlight beams from the regions to which the excitation light beams areapplied by the light irradiation units 210A and 210B.

The sample S is contained in a sample container such as a microplate ora slide glass, and is placed on the sample stage 290. The details of thesample stage 290 are the same as those of the sample stage 190 in thefirst embodiment.

The light irradiation units 210A and 210B respectively include lightsource units 211A and 211B, dichroic mirrors 222A and 222B, galvanomirrors 224A and 224B, objective lenses 226A and 226B, and objectivelens driving mechanisms 228A and 228B. The details of the light sourceunits 211A and 211B, dichroic mirrors 222A and 222B, galvano mirrors224A and 224B, objective lenses 226A and 226B, and objective lensdriving mechanisms 228A and 228B are the same as those of the lightsource unit 111, dichroic mirror 122, galvano mirror 124, objective lens126, and objective lens driving mechanism 128 in the first embodiment.

The photodetection units 230A and 230B respectively commonly include theobjective lenses 226A and 226B, the galvano mirrors 224A and 224B, andthe dichroic mirrors 222A and 222B with the light irradiation units 210Aand 210B. The photodetection units 230A and 230B also respectivelyinclude convergence lenses 232A and 232B, pinholes 234A and 234B,collimator lenses 236A and 236B, and photodetection units 231A and 231B.The convergence lenses 232A and 232B, the pinholes 234A and 234B, thecollimator lenses 236A and 236B, and the photodetection units 231A and231B are the same as the convergence lens 132, the pinhole 134, thecollimator lens 136, and the photodetection unit 131 in the firstembodiment.

The control unit 260 is constituted by, for example, a personalcomputer. The control unit 260 controls acquisition/storage/display offluorescence images, waiting for the input of settings of analysisregions, image analysis processing, the galvano mirrors 224A and 224B asan x-y scanning mechanism, the objective lens driving mechanisms 228Aand 228B as a z scanning mechanism, the sample stage 290, and the like.

FIG. 13 shows the functional blocks of the control unit shown in FIG.12. As shown in FIG. 13, the control unit 260 includes scan controlunits 262A and 262B, image forming units 264A and 264B, a storage unit266, a display unit 268, an input unit 270, an analysis processing unit276, and a stage control unit 280. The scan control units 262A and 262B,the image forming units 264A and 264B, the storage unit 266, and thestage control unit 280 constitute an analysis region image acquisitionunit in cooperation with the light irradiation units 210A and 210B andthe photodetection units 230A and 230B.

The scan control units 262A and 262B control the galvano mirrors 224Aand 224B so as to raster-scan the irradiation positions of excitationlight beams on observation regions in the sample S. In addition, ifnecessary, the scan control units 262A and 262B control the objectivelens driving mechanisms 228A and 228B so as to z-scan the irradiationpositions of excitation light beams on the sample S. The image formingunits 264A and 264B form fluorescence images of the sample S from theinformation of the irradiation positions of excitation light beams inputfrom the scan control units 262A and 262B and output signals from thephotodetection units 231A and 231B. With this operation, fluorescenceimages are acquired. The storage unit 266 stores the fluorescence imagesformed by the image forming units 264A and 264B. The display unit 268displays the fluorescence images of the sample S and analysis processingresults. The input unit 270 includes, for example, a mouse and akeyboard, and forms a GUI in cooperation with the display unit 268. ThisGUI is used to set, for example, an observation region. The stagecontrol unit 280 controls the sample stage 290 to set, for example, anobservation region in accordance with input information from the inputunit 270. The analysis processing unit 276 obtains a cross-correlationby using the data of pixels of images of two observation regionsacquired by the image forming units 264A and 264B.

The light irradiation units 210A and 210B, the photodetection units 230Aand 230B, the scan control units 262A and 262B, and the image formingunits 264A and 264B acquire fluorescence images of the respectiveobservation regions by the same technique as that for acquiringfluorescence images in the first embodiment. Fluorescence images of twoobservation regions may be acquired simultaneously or at differenttimings.

The analysis processing unit 276 performs image analysis processing inthe same manner as that for image analysis processing by the analysisprocessing unit 176 in the first embodiment except that the images oftwo analysis regions in the first embodiment are replaced by the imagesof two observation regions acquired by the image forming units 264A and264B.

This embodiment has the same advantages as those of the firstembodiment. In addition, simultaneously acquiring fluorescence images oftwo observation regions can obtain a more accurate spatialcross-correlation.

The embodiments of the invention have been described with reference tothe views of the accompanying drawing. The invention is not limited tothese embodiments and can be variously modified and changed within thespirit and scope of the invention.

For example, the embodiments have exemplified the analysis of imagesformed by detecting fluorescence emitted from the sample S, i.e.,fluorescence images.

However, images to be analyzed are not limited to fluorescence images.In addition to fluorescence images, images to be analyzed may include,for example, images formed by detecting phosphorescence, reflectedlight, visible light, chemiluminescence, bioluminescence, and scatteredlight.

The embodiments have exemplified images acquired by raster scanning.However, images to be processed are not limited to those acquired byraster scanning, and may be any images that are constituted by pixelswhose data are acquired time-serially, or may be images acquired byother scanning methods.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

What is claimed is:
 1. An image analysis apparatus comprising: ananalysis region image acquisition unit configured to acquire images ofspatially different analysis regions in a sample, each of the images ofthe analysis regions being constituted by pixels including a pluralityof data acquired time-serially, respectively; and a correlation analysisunit configured to obtain a cross-correlation between two analysisregions by using data of pixels of the images of the analysis regions,the analysis region image acquisition unit including first and secondoptical systems configured to simultaneously acquire images of theanalysis regions, respectively.
 2. The apparatus of claim 1, wherein theanalysis region image acquisition unit comprising: an observation regionimage acquisition unit configured to acquire an image of an observationregion including the analysis regions, the image of the observationregion being constituted by pixels including a plurality of dataacquired time-serially, respectively; an analysis region setting unitconfigured to set the analysis regions for the image of the observationregion; and a data extraction unit configured to extract data of pixelscorresponding to the analysis regions from the image of the observationregion.
 3. The apparatus of claim 2, wherein the observation regionimage acquisition unit acquires images of frames of the observationregion.
 4. The apparatus of claim 3, wherein the correlation analysisunit obtains the cross-correlation by using any one of a fluorescenceintensity, a pixel time, a line time, a frame time, a pixel positionalrelationship, a pixel size, and a statistical value thereof.
 5. Theapparatus of claim 1, wherein the analysis region image acquisition unitacquires images of frames of the analysis regions.
 6. The apparatus ofclaim 1, wherein the correlation analysis unit obtains thecross-correlation by using any one of a fluorescence intensity, a pixeltime, a line time, a frame time, a pixel positional relationship, apixel size, and a statistical value thereof.
 7. The apparatus of claim1, wherein the correlation analysis unit obtains the cross-correlationby using any one of an average value, a maximum value, a minimum value,a relative difference, and an absolute difference of the data.
 8. Theapparatus of claim 1, wherein the correlation analysis unit obtains thecross-correlation by using reconstructed data obtained by reconstructingthe data.
 9. The apparatus of claim 1, wherein each image of theanalysis regions is a two-dimensional image, the two analysis regions,between which the cross-correlation is obtained, are an analysis region1 and an analysis region 2, and the correlation analysis unit performscorrelation calculation by using equation (1) and fitting for thecorrelation calculation result by using equation (2) to obtain thecross-correlation between the two-dimensional analysis regions:$\begin{matrix}{{G_{s}\left( {\xi,\psi} \right)} = \frac{\Sigma \; {I_{1}\left( {x,y} \right)}*{{I_{2}\left( {{x + \xi},{y + \psi}} \right)}/M_{12}}}{\left( {\Sigma \; {{I_{1}\left( {x,y} \right)}/M_{1}}} \right)\left( {\Sigma \; {{I_{2}\left( {x,y} \right)}/M_{2}}} \right)}} & (1)\end{matrix}$ where G_(s) is a spatial cross-correlation value of RICS,I₁ is data of a pixel of an image of the analysis region 1, I₂ is dataof a pixel of an image of the analysis region 2, x and y are the spatialcoordinates of a measurement point, ξ and ψ are the changes of thespatial coordinates from the measurement point, M₁₂ is the number oftimes of sum-of-product calculation of data of the analysis region 1 andthe analysis region 2, M₁ is the total number of data of the analysisregion 1, and M₂ is the total number of data of the analysis region 2$\begin{matrix}{\mspace{79mu} {{{G_{s}\left( {\xi,\psi} \right)} = {{S\left( {\xi,\psi} \right)}*{G\left( {\xi,\psi} \right)}}}\mspace{20mu} {{S\left( {\xi,\psi} \right)} = {\exp \left( {- \frac{\frac{1}{2}*\left\lbrack {\left( \frac{2{\xi\delta}_{r}}{W_{0}} \right)^{2} + \left( \frac{2{\psi\delta}_{r}}{W_{0}} \right)^{2}} \right\rbrack}{\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi}} \right)}}{W_{0}^{2}}} \right)}} \right)}}{{G\left( {\xi,\psi} \right)} = {\frac{1}{N}\left( {\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi}} \right)}}{W_{0}^{2}}} \right)^{- 1}*\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi}} \right)}}{W_{z}^{2}}} \right)^{{- 1}/2}} \right)}}}} & (2)\end{matrix}$ where G_(s) is a spatial correlation value of RICS, S isan influence of a scan in RICS analysis, G is an influence of a timedelay in RICS analysis, D is a diffusion constant, δ_(r) is a pixelsize, N is the number of molecules, ξ and ψ are the changes of spatialcoordinates from a measurement point, W₀ is a radius of an excitationlaser beam in a lateral direction, W_(z) is a radius of an excitationlaser beam in a longitudinal direction, τ_(p) is a pixel time, and τ_(l)is a line time.
 10. The apparatus of claim 1, wherein each image of theanalysis region is a three-dimensional image, the two analysis regions,between which the cross-correlation is obtained, are an analysis region1 and an analysis region 2, and the correlation analysis unit performscorrelation calculation by using equation (3) and fitting for thecorrelation calculation result by using equation (4) to obtain thecross-correlation of the three-dimensional analysis regions:$\begin{matrix}{{G_{s}\left( {\xi,\psi,\eta} \right)} = \frac{\Sigma \; {I_{1}\left( {x,y,z} \right)}*{{I_{2}\left( {{x + \xi},{y + \psi},{z + \eta}} \right)}/M_{12}}}{\left( {\Sigma \; {{I_{1}\left( {x,y,z} \right)}/M_{1}}} \right)\left( {\Sigma \; {{I_{2}\left( {x,y,z} \right)}/M_{2}}} \right)}} & (3)\end{matrix}$ where G_(s) is a spatial cross-correlation value of RICS,I₁ is data of a pixel of an image of the analysis region 1, I₂ is dataof a pixel of an image of the analysis region 2, x, y, and z are thespatial coordinates of a measurement point, ξ, ψ, and η are the changesof the spatial coordinates from the measurement point, M₁₂ is the numberof times of sum-of-product calculation of data of the analysis region 1and the analysis region 2, M₁ is the total number of data of theanalysis region 1, and M₂ is the total number of data of the analysisregion 2 $\begin{matrix}{\mspace{79mu} {{{G_{s}\left( {\xi,\psi,\eta} \right)} = {{S\left( {\xi,\psi,\eta} \right)}*{G\left( {\xi,\psi,\eta} \right)}}}\mspace{20mu} {{S\left( {\xi,\psi,\eta} \right)} = {\exp \left( {- \frac{\frac{1}{2}*\left\lbrack {\left( \frac{2{\xi\delta}_{r}}{W_{0}} \right)^{2} + \left( \frac{2{\psi\delta}_{r}}{W_{0}} \right)^{2} + \left( \frac{2{\eta\delta}_{r}}{W_{0}} \right)^{2}} \right\rbrack}{\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi} + {\tau_{f}\eta}} \right)}}{W_{0}^{2}}} \right)}} \right)}}{{G\left( {\xi,\psi,\eta} \right)} = {\frac{1}{N}\left( {\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi} + {\tau_{f}\eta}} \right)}}{W_{0}^{2}}} \right)^{- 1}*\left( {1 + \frac{4\; {D\left( {{\tau_{p}\xi} + {\tau_{l}\psi} + {\tau_{f}\eta}} \right)}}{W_{z}^{2}}} \right)^{{- 1}/2}} \right)}}}} & (4)\end{matrix}$ where G_(s) is a spatial correlation value of RICS, S isan influence of a scan in RICS analysis, G is an influence of a timedelay in RICS analysis, D is a diffusion constant, δ_(r) is a pixelsize, N is the number of molecules, ξ, ψ, and η are the changes ofspatial coordinates, W₀ is a radius of an excitation laser beam in alateral direction, W_(z) is a radius of an excitation laser beam in alongitudinal direction, τ_(p) is a pixel time, τ_(l) is a line time, andτ_(f) is a frame time.
 11. The apparatus of claim 1, wherein theanalysis region image acquisition unit includes first and second lightirradiation units configured to apply light beams to measurement pointsin the analysis regions, respectively.
 12. The apparatus of claim 1,wherein the analysis region image acquisition unit includes first andsecond photodetection units configured to detect light beams frommeasurement points in the analysis regions, respectively.
 13. Theapparatus of claim 1, wherein the analysis region image acquisition unitincludes first and second light irradiation units configured to applylight beams to measurement points in the analysis regions, respectively,and first and second photodetection units configured to detect lightbeams from the measurement points in the analysis regions, respectively.14. The apparatus of claim 13, wherein each of the first and secondlight irradiation units includes a light source unit, a dichroic mirror,a galvano mirror, and an objective lens on its optical passes.
 15. Theapparatus of claim 14, wherein each of the first and second lightirradiation units includes a driving mechanism configured to drive thecorresponding objective lens.
 16. The apparatus of claim 13, whereineach of the first and second photodetection units includes an objectivelens, a galvano mirror, a convergence lens, a collimator lens, and aphotodetection unit on its optical pass.
 17. The apparatus of claim 14,wherein the first photodetection unit includes the objective lens,galvano mirror, and dichroic mirror in the first light irradiation uniton its optical pass, and the second photodetection unit includes theobjective lens, galvano mirror, and dichroic mirror in the second lightirradiation unit on its optical pass.
 18. The apparatus of claim 15,wherein each of the galvano mirror scans the corresponding measurementpoint in directions perpendicular to an optical axis of thecorresponding objective lens, each of the objective lens scans thecorresponding measurement point in a direction parallel to the opticalaxis.